{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "429d65c8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 8.68 ms (started: 2021-08-13 03:39:57 +08:00)\n"
     ]
    }
   ],
   "source": [
    "# 自动计算cell的计算时间\n",
    "%load_ext autotime\n",
    "\n",
    "%config InlineBackend.figure_format='svg' #矢量图设置，让绘图更清晰\n",
    "\n",
    "# 导入tensorboard\n",
    "%load_ext tensorboard"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "28476536",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "origin\tgit@github.com:ustchope/kerastuner.git (fetch)\n",
      "origin\tgit@github.com:ustchope/kerastuner.git (push)\n",
      "[master 9a5e7ff] 更新 #1 Aug 13, 2021\n",
      " 2 files changed, 132 insertions(+), 31 deletions(-)\n",
      " create mode 100644 \"\\345\\217\\257\\350\\247\\206\\345\\214\\226\\350\\266\\205\\345\\217\\202\\346\\225\\260\\350\\260\\203\\346\\225\\264\\350\\277\\207\\347\\250\\213.ipynb\"\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "To github.com:ustchope/kerastuner.git\n",
      "   2a9432b..9a5e7ff  master -> master\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 3.69 s (started: 2021-08-13 02:39:20 +08:00)\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "# 增加更新\n",
    "git add *.ipynb\n",
    "\n",
    "git remote -v\n",
    "\n",
    "git commit -m '更新 #1 Aug 13, 2021'\n",
    "\n",
    "git push origin master"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "d3e426ec",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 1.47 s (started: 2021-08-13 02:39:38 +08:00)\n"
     ]
    }
   ],
   "source": [
    "#设置使用的gpu\n",
    "import tensorflow as tf\n",
    "\n",
    "gpus = tf.config.list_physical_devices(\"GPU\")\n",
    "\n",
    "if gpus:\n",
    "   \n",
    "    gpu0 = gpus[0] #如果有多个GPU，仅使用第0个GPU\n",
    "    tf.config.experimental.set_memory_growth(gpu0, True) #设置GPU显存用量按需使用\n",
    "    # 或者也可以设置GPU显存为固定使用量(例如：4G)\n",
    "    #tf.config.experimental.set_virtual_device_configuration(gpu0,\n",
    "    #    [tf.config.experimental.VirtualDeviceConfiguration(memory_limit=4096)]) \n",
    "    tf.config.set_visible_devices([gpu0],\"GPU\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c96d0a45",
   "metadata": {},
   "source": [
    "# 介绍"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d05e8f66",
   "metadata": {},
   "source": [
    "KerasTuner 将日志打印到屏幕上，包括每次试验中的超参数值，供用户监控进度。 然而，阅读日志不够直观，无法感知超参数对结果的影响，因此，我们提供了一种使用 TensorBaord 通过交互式图形可视化超参数值和相应评估结果的方法。\n",
    "\n",
    "TensorBoard 是用于可视化机器学习实验的有用工具。 它可以在模型训练期间监控损失和指标，并可视化模型架构。 使用 TensorBoard 运行 KerasTuner 将为您提供使用其 HParams 插件可视化超参数调整结果的附加功能。\n",
    "\n",
    "我们将使用一个为 MNIST 图像分类数据集调整模型的简单示例来展示如何将 KerasTuner 与 TensorBoard 结合使用。\n",
    "\n",
    "第一步是下载和格式化数据。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "a280139b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(60000, 28, 28, 1)\n",
      "(60000,)\n",
      "(10000, 28, 28, 1)\n",
      "(10000,)\n",
      "time: 767 ms (started: 2021-08-13 02:46:38 +08:00)\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import keras_tuner as kt\n",
    "from tensorflow import keras\n",
    "from tensorflow.keras import layers\n",
    "\n",
    "(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()\n",
    "# Normalize the pixel values to the range of [0, 1].\n",
    "x_train = x_train.astype(\"float32\") / 255\n",
    "x_test = x_test.astype(\"float32\") / 255\n",
    "# Add the channel dimension to the images.\n",
    "x_train = np.expand_dims(x_train, -1)\n",
    "x_test = np.expand_dims(x_test, -1)\n",
    "# Print the shapes of the data.\n",
    "print(x_train.shape)\n",
    "print(y_train.shape)\n",
    "print(x_test.shape)\n",
    "print(y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e391a5e7",
   "metadata": {},
   "source": [
    "然后，我们编写一个 `build_model` 函数来使用超参数构建模型并返回模型。 超参数包括要使用的模型类型（多层感知器或卷积神经网络）、层数、单元或过滤器的数量、是否使用 dropout。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "e64964f2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 1.95 ms (started: 2021-08-13 02:49:52 +08:00)\n"
     ]
    }
   ],
   "source": [
    "def build_model(hp):\n",
    "    inputs = keras.Input(shape=(28, 28, 1))\n",
    "    # Model type can be MLP or CNN.\n",
    "    model_type = hp.Choice(\"model_type\", [\"mlp\", \"cnn\"])\n",
    "    x = inputs\n",
    "    if model_type == \"mlp\":\n",
    "        x = layers.Flatten()(x)\n",
    "        # Number of layers of the MLP is a hyperparameter.\n",
    "        for i in range(hp.Int(\"mlp_layers\", 1, 3)):\n",
    "            # Number of units of each layer are\n",
    "            # different hyperparameters with different names.\n",
    "            output_node = layers.Dense(\n",
    "                units=hp.Int(f\"units_{i}\", 32, 128, step=32), activation=\"relu\",\n",
    "            )(x)\n",
    "    else:\n",
    "        # Number of layers of the CNN is also a hyperparameter.\n",
    "        for i in range(hp.Int(\"cnn_layers\", 1, 3)):\n",
    "            x = layers.Conv2D(\n",
    "                hp.Int(f\"filters_{i}\", 32, 128, step=32),\n",
    "                kernel_size=(3, 3),\n",
    "                activation=\"relu\",\n",
    "            )(x)\n",
    "            x = layers.MaxPooling2D(pool_size=(2, 2))(x)\n",
    "        x = layers.Flatten()(x)\n",
    "\n",
    "    # A hyperparamter for whether to use dropout layer.\n",
    "    if hp.Boolean(\"dropout\"):\n",
    "        x = layers.Dropout(0.5)(x)\n",
    "\n",
    "    # The last layer contains 10 units,\n",
    "    # which is the same as the number of classes.\n",
    "    outputs = layers.Dense(units=10, activation=\"softmax\")(x)\n",
    "    model = keras.Model(inputs=inputs, outputs=outputs)\n",
    "\n",
    "    # Compile the model.\n",
    "    model.compile(\n",
    "        loss=\"sparse_categorical_crossentropy\", metrics=[\"accuracy\"], optimizer=\"adam\",\n",
    "    )\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd78cd91",
   "metadata": {},
   "source": [
    "我们可以对模型进行快速测试，以检查它是否为 CNN 和 MLP 成功构建。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "f925f8fd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_1 (InputLayer)         [(None, 28, 28, 1)]       0         \n",
      "_________________________________________________________________\n",
      "conv2d (Conv2D)              (None, 26, 26, 32)        320       \n",
      "_________________________________________________________________\n",
      "max_pooling2d (MaxPooling2D) (None, 13, 13, 32)        0         \n",
      "_________________________________________________________________\n",
      "flatten (Flatten)            (None, 5408)              0         \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (None, 10)                54090     \n",
      "=================================================================\n",
      "Total params: 54,410\n",
      "Trainable params: 54,410\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "Model: \"model_1\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_2 (InputLayer)         [(None, 28, 28, 1)]       0         \n",
      "_________________________________________________________________\n",
      "flatten_1 (Flatten)          (None, 784)               0         \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 10)                7850      \n",
      "=================================================================\n",
      "Total params: 7,850\n",
      "Trainable params: 7,850\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "time: 5.43 s (started: 2021-08-13 02:52:14 +08:00)\n"
     ]
    }
   ],
   "source": [
    "# Initialize the `HyperParameters` and set the values.\n",
    "hp = kt.HyperParameters()\n",
    "hp.values[\"model_type\"] = \"cnn\"\n",
    "# Build the model using the `HyperParameters`.\n",
    "model = build_model(hp)\n",
    "# Test if the model runs with our data.\n",
    "model(x_train[:100])\n",
    "# Print a summary of the model.\n",
    "model.summary()\n",
    "\n",
    "# Do the same for MLP model.\n",
    "hp.values[\"model_type\"] = \"mlp\"\n",
    "model = build_model(hp)\n",
    "model(x_train[:100])\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a4362a1",
   "metadata": {},
   "source": [
    "使用 10 次试验初始化 RandomSearch 调谐器，并使用验证准确性作为选择模型的指标。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "8ce1696f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 144 ms (started: 2021-08-13 02:55:37 +08:00)\n"
     ]
    }
   ],
   "source": [
    "tuner = kt.RandomSearch(\n",
    "    build_model,\n",
    "    max_trials=10,\n",
    "    # Do not resume the previous search in the same directory.\n",
    "    overwrite=True,\n",
    "    objective=\"val_accuracy\",\n",
    "    # Set a directory to store the intermediate results.\n",
    "    directory=\"/tmp/tb\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b61d6c30",
   "metadata": {},
   "source": [
    "通过调用 `tuner.search(...)` 开始搜索。 要使用 TensorBoard，我们需要将 `keras.callbacks.TensorBoard` 实例传递给回调。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "d6f94f42",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Trial 10 Complete [00h 00m 22s]\n",
      "val_accuracy: 0.9205833077430725\n",
      "\n",
      "Best val_accuracy So Far: 0.9838333129882812\n",
      "Total elapsed time: 00h 03m 47s\n",
      "INFO:tensorflow:Oracle triggered exit\n",
      "time: 3min 47s (started: 2021-08-13 02:59:02 +08:00)\n"
     ]
    }
   ],
   "source": [
    "tuner.search(\n",
    "    x_train,\n",
    "    y_train,\n",
    "    validation_split=0.2,\n",
    "    epochs=2,\n",
    "    # Use the TensorBoard callback.\n",
    "    # The logs will be write to \"/tmp/tb_logs\".\n",
    "    callbacks=[keras.callbacks.TensorBoard(\"/tmp/tb_logs\")],\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b5cf0dc6",
   "metadata": {},
   "outputs": [],
   "source": [
    "如果在 Colab 中运行，以下两个命令将显示 Colab 中的 TensorBoard。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "44cf6b12",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Reusing TensorBoard on port 6007 (pid 942281), started 0:00:13 ago. (Use '!kill 942281' to kill it.)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "      <iframe id=\"tensorboard-frame-73db933b4f6cc7d3\" width=\"100%\" height=\"800\" frameborder=\"0\">\n",
       "      </iframe>\n",
       "      <script>\n",
       "        (function() {\n",
       "          const frame = document.getElementById(\"tensorboard-frame-73db933b4f6cc7d3\");\n",
       "          const url = new URL(\"/\", window.location);\n",
       "          const port = 6007;\n",
       "          if (port) {\n",
       "            url.port = port;\n",
       "          }\n",
       "          frame.src = url;\n",
       "        })();\n",
       "      </script>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 8.33 ms (started: 2021-08-13 03:41:46 +08:00)\n"
     ]
    }
   ],
   "source": [
    "%tensorboard --logdir /tmp/tb_logs --port 6007"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0463b9a",
   "metadata": {},
   "source": [
    "您可以访问 TensorBoard 的所有常见功能。 例如，您可以查看损失和指标曲线，并可视化不同试验中模型的计算图。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96f0f2cf",
   "metadata": {},
   "source": [
    "除了这些功能之外，我们还有一个 HParams 选项卡，其中包含三个视图。 在表格视图中，您可以查看具有不同超参数值和评估指标的表格中的 10 个不同试验。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "506247ba",
   "metadata": {},
   "source": [
    "在左侧，您可以为某些超参数指定过滤器。 例如，您可以指定仅查看没有 dropout 层和 1 到 2 个密集层的 MLP 模型。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a4d2784",
   "metadata": {},
   "source": [
    "除了表格视图，它还提供了另外两个视图，平行坐标视图和散点图矩阵视图。 它们只是相同数据的不同可视化方法。 您仍然可以使用左侧的面板来过滤结果。\n",
    "\n",
    "在平行坐标视图中，每条彩色线都是一个试验。 轴是超参数和评估指标。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ea46bc8",
   "metadata": {},
   "source": [
    "在散点图矩阵视图中，每个点都是一个试验。 这些图是试验在以不同超参数和度量为轴的平面上的投影。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "75eb4d07",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:tf2keras]",
   "language": "python",
   "name": "conda-env-tf2keras-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
